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The, Validation of learning Hierarchies 



One of the-most rapidly growing areas in the field of .education today 
is the area of individualized instruction-- instruction in which content 
organization, or pacing is modified for each individual . Programs of ' 
individualized instruction have been implemented in such areas as electronics 
Peiper Swegey and Valverde, 1970), engineering (Schure, 1965), mathematics 
(Bushnell, 1966), psychology (Kulik, 1972), and vocational and technical 
education in the military (Impelliterri and Finch, 1971). These programs 
employ several different types of educational technology', including personalized 
systems of instruction (PSI), computer-assisted instruction (CM), individually 
prescribed instruction (IPI), and programmed instruction (PI). 

Although individualized instruction comes in many forms, all of these 
forms shar* the same basic design. All are essentially sequences of instruc- 
tional units through which subjects are routed by means of a series of 
te^ts. The way in which these units are sequenced is one of- the more 
crucial components of any individualized instruction program. The units 
should be arranged in such a fashion that prerequisite material is covered 
first, followed by more advanced material. How this ordering is determined 
has been the topic of considerable research. The purpose of this report is 
to evaluate .the various approaches to sequencing instructional units that M 
have been reported in the literature, and to make recommendations on the 
direction future research should follow. 



Most of the procedures that have been proposed for sequencing instruc- 
tional units have been based on v the work of Gagne. (1962), and have involved 
the use of various techniques for validating learning hierarchies' constructed 
using his task analysis methodology. Procedures fbr validating hierarchies 
have generally taken one of two approaches One approach has been to 
simply compute a coefficient that measures the strength of the hierarchical 
relationships, and on the basis of the coefficient accept or reject the 
hierarchy. The other approach has been to more completely describe the 
relationships among the instructional units, usually by means* of a mathematical 
model and to decide on the basis of the description whether the relationships 
are of the desired nature. Both approaches will be discussed-, as will be a 
number of examples of each approach. First,- however, a discussion of 
Gagne s work on learning hierarchies will be presented. 

The Work of Gagne 

In a series of articles dating from 1961, Gagne\and his coworkers 
investigated the learning of mathematical skills. A hypothesis .originating 
from this research was that a class of tasks necessary to attain a learning 
outcome could be sorted into a hierarchy of sets of tasks, each set being a 
group of tasks at the same level in the hierarchy.. The structure of the 
hierarchy would be such that there could be positi-ve transfer of learning 
from one learning set to a higher level learning set until the final learning 
outcome v<as achieved. Gagne (1962) proposed that one could establish a 
learning_hierarchy by determining the skills an individual would have to 
possess in order to attain the learning outcome. One would then make the 
same determination for each of those skills. This process would be continued 



until one developed a hierarchy containing the most basjc skills' as the / 
learning get at the lowest* level. If the established hierarchy , were eorrett, 
then., positive transfer from each lower learning set to the next higher ^ 
•learning set would be promoted (Gagne* 1962). 4 

9 U 

The successful establishment of a learning hierarchy result's in a> 
vertical 'structure of learning sets, with any high level learning set 
having one or more immediately prerequisite learning sets. Figure 1' presents 
an example of a derived learning hierarchy. In this hierarchy, the terminal 
task has three prerequisite tasks, which in turn have six second level 
prerequisite tasks. Mastery of the* lower tasks is required before a final 
task can be performed. „ , 

* x * Figure 1 

An Example of a Derived Hierarchy 



Final 
Task 



First Level Set 



Second Level Set I J * _J - |^ [ . 

\> 

Gagne (1968) identified two characteristics necessary for the successful 
establishment of a learning hierarchy. One such characteristic is that of 
sequencing: a learner who is able to perform successfully a higher level 
set of tasks .should also be able to perform aU lower level sets of tasks 
in the hierarchy. The other characteristic is transfer: attainment of a 
lower lev'el c learning outcome should increase the probability of successful 
attainment of a higher level learning set. The existence of these two 
characteristics establishes that there is an ordered relationship among the 
learning sets within the hierarchy. If a learning hierarchy has been 
successfully established, it should be possible to validate the hierarchy 
by demonstrating the presence of these characteristics. Most of the procedures 
for validating hierarchies, including Gagne' s own procedure, are based on 
this premise. Those procedures will now be discussed. 

Proce dures for 'Va li dating Hierarchies 

As^as previously mentioned, there have been two basic approaches to 
validating hierarchies constructed using task analysis procedures, one 
based on coefficients and one based on more complete descriptions 0/ the 
relationships among sets of learning tasks. These two approaches will now 
be discussed in more detail, and several examples of each will be presented. 
After the approaches and examples have been discussed, an evaluation of the 
approaches will be presented. 
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Co ef ficient-Bised Procedures 

I . 
.The basic objective o£_the coefficient approach for validating learning 
hierarchies is to compute the value of a coefficient that will measiire the 
strength' of the ordered relationships among learning sets or tasks A 
number of coefficients have been proposed for this purpose, most of which 
are variations on. the proportion of positive transfer (PPT) statistic 
proposed by Gagne" and Paradise (1961). The PPT statistic will be described 
first, and then the ways in which it has been modified will be discussed. 
Afterward, several coefficients not based on the PPT statistic will be 
discussed. 

The_PPT Statistic The PPT statistic is based on the four pass/fail 
combinations possible for two learning tasks. If Task 2 follows (is dependent 
op) Task 1 and N 0 o is the number of subjects failing on both tasks, N 10 is 
the number of subjects succeeding on Task 1 but failing on Task 2, 'and N lt 
is the number of subjects succeeding on both tasks, then the PPT statistic 
is given by 

N 00 + N n 
PPT = — l — 



Noo +' N 10 + Nn 



This statistic is used to measure the level of positive transfer from Task 
1 to Task 2. If the level of positive transfer is high, then fewer subjects 
will succeed on Task 1 without succeeding on Task 2 than would haVe if Xhe 
level of transfer had been low. Thus, the greater the positive transfer 
the smaller is N 10 . 'As N 10 decreases, the PPT statistic increases. When * 
N 10 is 0, the PPT statistic has a value of 1.0, ,whith is the maximum value 
it can take on. 

♦ 

The PPT statistic appears to be a reasonable mdjcant of the level of 
positive transfer between two tasks. However, positive trans fer, does not 
necessarily indicate that there is a hierarchical relationship between the 
two tasks. It is possible that the positive transfer from Task 2 to Task 1 . 
is as great as the positive transfer from Task 1 to Task 2. The PPT statistic 
does not indicate the direction of the positive transfer. In fact if the 
tasks are reversed so that Task 2 becomes the first task' and Task 1 becomes 
tne second task, an eqial or higher value may be obtained for the PPT 
statistic as was obtained before the reversal, but in this situation the 
statistic indicates positive transfer from the subsequent task to» the 
precedent task. Because of this limitation on the interpretation of the 

V t ta lll tlS ' 3 aUmber ° f alternativ e statistics, most of which are variations 
ot tile PPI statistic, have been proposed. 

> Variations on the PPT Statistic The Commission on Science "Education > 
of the Amarican Association for the Advancement of Science (Walbesser, 
1968) proposed a procedure for validating hierarchies that employs three 
statistics rather than one. These three statistics are the consistency, 
the adequacy, and the completeness ratios. These ratios use the 'same four 
pass/fail combinations that were used for the PPT statistic. 



O 



The consistency ratio (CR) is given by 

CR = ■ , 



Nu + N 10 



where the terms are as defined for the PPT, statistic As can be seen, the 
CR statistic is tfie same as the PPT statistic, except that the N 0 o term is 
omitted. The Nqo term is omitted because, while it is consistent with the 
■hypothesized hierarchical relationship, it is not an indication of positive 
'transfer (Walbesser and Eisenberg, 1972). 

Consistency is a necessary but not sufficient condition for a valid 
hierarchy, according to Walbesser (1968), In addition, adequacy and 
completeness must be considered. Adequacy refers to^how often the learneV , 
Has achieved a behavior ?fter the relevant subordinate behavior has been 
attained. The adequacy of a hierarchy is measured using the adequacy ratio 
(AR),^which is given by *» 

• N n , ' \ 

' AR = . 



Nu + Hoi 



Completeness is an indication of the number of examinees that have 
reached the terminal behavior relative t(o those -th^t have made no progress. 
High consistency and adequacy ratios are misleading if enly small numbers 
of examinees acquire*the terminal behavior and at least some subordinate 
behaviors. Thus, a Large value for Noo would be evidence of incomplete 
, , instruction- The completeness ratio (COR) is given by 

' Mil 

* COR = — 

- ' V- Jl}r* + N 0 o 

tf 

A-procedure far validating hierarchies proposed by Walbesser and 
Eisenberg (1971) employs the v consistency, adequacy, and completeness ratios, 
and in addition uses two other coefficients. These coefficients are the 
inverse consistency ratio (ICR) and the inverse adequacy ratio (IAR). 
>.wftile consistency indicates that the acquisition' of the terminal behavior 
implies the acquisition of subordinate behaviors, the inverse consistency 
ratio meastrrts the extent to which nohacquisition of the terminal behavior 
( implies nonacquisition of subordinate behaviors. The inverse consistency 

ratio is given by * ■ ^ 

ICR = . 
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Noo + N 01 , 

' The adequacy ratio measures the extent to which the acquisition of all 
Subordinate' behaviors implies^ acquisition of the terminal behavior, while 
the inverse adequacy ratio i/idicates the degree to which nonacquisition of 
the subordinate behavior implies nonacquisition of the terminal behavior. 
The N inverse adequacy ratio is given by 

N 00 ' 
\ IAR = . 

Noo"* N 10 • • 

. ' . ■ 9- . .* 
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Non-PPT Ba sed Statistics Not all, of the coefficients that , have been 
proposed for validating hierarchies have been variations on the PPT statistic 
For instance, Capie and Jones (1971) used the phi coefficient for validating 
hierarchies. The phi coefficient is based on the sajne four pass/fail 
combinations that were used f6r the PPT-like statistics. However, rather 
than computing a ratio of the numbers of subjects in specified groups, a 
product ^moment correlation coefficient is computed. 

tr 

The Guttman coefficient of reproducibility has also been used to 
validate hierarchies (Hofman, 1977; ResniCk and Wang, 1969). An assumption 
made when this statistic is used is that the learning tasks are ordered- 
according to their difficulty. If an individual gives five correct responses 
on 10 items, it is assumed that the individual responded correctly to the 
five easiest items. This statistic, T:hen, i's a measure of the relationship 
between the response pattern of an individual and the number of correct 
responses by that individual. If the response pattern is not accurately 
predicted by the number of correct responses', a perfect Guttman scale is 
not present. The proportion of responses that follow the predicted pattern 
is called the coefficient of reproducibility. To the extent that some 
respondents achieve success on some^of the more difficult tasks but fail on 
less difficult tasks, the coefficient of reproducibility is /educed in 
magnitude. For a pair of tasks in a hierarchy, the extent to which some, 
respondents achieve success on the superordinate task but fail the sub- 
ordinate task, the coefficient is reduced. "-TheS^f f icient>Kreproducibility< 
for the hierarchy is the average coefficient of reproducibility) for all of 
the pairs of superordinate and subordinate tasks ^n the hierarchy (Hofman 
1977). . • ' 

Another statistic that h"as been used for validating hierarchies is the 
proportion of disconf irmatory response patterns (Airasian and Bart, 1975). 
This procedure is based on ordering theory, which provides a basis for 
determining logical relationships among tasks (Bart and Krus, 1973). If it 
is assumed that Task 1 is' prerequisite to Task 2, then the response pattern 
(.0,1), which indicates success on Task 2 but failure on Task 1, is considered 
disconfirmatory. The response patterns' (0,0), (1,0), and (1,1) are considered 
to be confirmatory. The proportion of disconfirmatory response patterns 
(PD) is given by 

Noi' ' * , 

PD = — 



Noo + N 10 + N 01 + N u 



One other statistic that has been used for validating hierarchies is 
the conditional item difficulty index (Mrasian, 1971). The conditional 
difficulty of an item is computed using only the -subjects having response 
patterns that^e predicted from the hierarchy. For instance, *or a three- 
task hierarchy the only response patterns . expected are: * 

(a) 000, 

(b) 100, . 

(c) 110, V. 
and (d) 111^ 



iU 



*6- 



even though there are eight patterns possible. If no, n 1? n2 , and 113 are 
the number of 'subjects with response patterns (a), ,(b) , (c), and (d) above, 
respectively, then only no + n\ + n% + n3 subjects will be considered when 
computing a conditional item difficulty. The conditional difficulty for an 
'item is given by the number of subjects having expected response patterns 
in which the item was correctly answered, divided by the number of subjects 
having expected response patterns in which all preceding items are correct. 
Thus for the first task in the three-task hierarchy the conditional difficulty 
(CD) is given by 

n i + n 2 + n 3 

■ CD = 



For the second task the conditional difficulty is given by 

n 2 + n 3 

cu = , * ; 



and for the third task the conditional difficulty is given by 

n 3 ' 



CD 



n 2 + n 3 



The conditional item difficulty irtdices .computed for a set of tasks 
fielp to determine the validity of a hierarchy by indicating the extent to 
which failure on earlier tasks is predictive of failure on later tasks. 
For example, if the conditional difficulty of an item is very low given 
that all preceding items have been correctly answered, the completeness of 
the sequence is questionable (Airasian, 1971). 

Summary of Coefficient Procedures The basic^ objective of the coefficient 
approach to validating hierarchies is to compute a coefficient that will 
indicate the strength of the hierarchical relationships among learning 
^tasks. Many different coefficients have been proposed for this purpose, 
including: .the proportion of positive transfer statist ic ; e the consistency, 
adequacy , corfipleteness , inverse ^consistency, and inverse adequacy ratios;* 
the* Guttman coefficient of reproducibility; the phi coefficient; the proportion 
, o£ disconf irmatory response patterns; and the conditional item difficulty 
x index. Table 1 summarizes the coefficients that have been discussed. •. - 

V ' . 

One reason why so many coefficients have been proposed is that~each 
coefficient tends to be associated with only one a*spect of the complex 
relationship between tasks in a hierarchy. This has led some researchers 
to propose the use of "multiple coefficients for the ^validation of hierarchies 
(Walbesser, 1968; Walbesser and Edsenberg, 1972). Other researchers have- 
proposed that more complete descriptions of the relationship betweeri learning 
tasks is needed than can be provided by coefficients alone. These researchers 
have tended to use mathematical models »to describe the relationships among 
learning tasks. This^ approach to validating learning hierarchies will be 
discussed next . * 

« 



Model-Based Procedures 



Model-based procedures specify a mathematical model to describe the 
relationship between performances, on different tasks in a hierarchy This 
is commonly done using a mathematical expression describing the probability 
of success on a task given the performance (acquisition Vs. nonacquisition) 
on lower level tasks. The adequacy of the model for describing the relationship 
between performances on different tasks can be tested statistically if 
the performance on a task predicted from the model does not differ significantly 
.from observed data, then the data support the hypothesized structure of the 
hierarchy and Che form of the mathematical model. If the model does not 
fit the data, the failure may be due to errors in the hypothesized structure 
of the hierarchy, errors in the form of the mathematical model, or both. 

Proctor Model One of the first mathematical models proposed for 
validating learning hierarchies was described by Proctor (1970) This 
model is a probabilistic formulation of Guttman scaling. The items in the 
proposed hierarchy are assumed to form a Guttman scale. Each response 
pattern consistent with the Guttman scale is associated with a true ahilitv 
level It is assumed that patterns that are inconsistent with the scale 
are also associated with one of the true ability levels, but that they 
cqntaxn error. The probability that an observed response pattern is' assocated 
with a given true ability level is the probability of finding an examinee 
" true ability level multiplied by the probability that an examinee 

with that ability level would give the observed response pattern This 
second probability decreases as the difference between the observed pattern 
and the pure pattern associated with that level of. ability increases The 
probability of the occurrence of an observed response pattern is the sum Sf 
the probabilities of-the pattern over each of the true ability levels. ' 

As an example, suppose that every subject in a population belongs to 
one of the several .ability levels, each of which has associated with it a 
true Guttman type response pattern. For a three item scale there are four 
true Guttman patterns-- (000) , (100), (110), and (111). Every subject, 
then, belongs to pne of the four ability levels associated with these 
patterns. The probability of an observed response pattern for this scale 
is the sum of four terms- the probability of finding' a subject whose true 
ability level was/associated with the (000) pattern and who responded with 
the observed pattern, the probability of finding a subject who should have, 
respongd With the (100) pattern but,wh Q responded wi*h the observed pattern 
and so^ until all four true patterns have been considered. In its mathe- ' 
matical form, the probability of the observed pattern for the three item 
scale is given by 

«V - ^ - »'\ ^ 2 n - -> yV »" 3 <i - « >"" 3 , ♦ -V- . (i) 

where x is the observed pattern; 6„ 0 2 , 0 3 , and 0, are the proportions of 
the population having each true ability level; the a term is the probability 
of a subject responding to an item in a way inconsistent with the true * * 

pattern assorted with the subject's true ability; and the n,, n 2 \ n 3f and 
n4 terms are the number of items in the observed pattern inconsistent with 
•the true pattern for each of the four ability levels. 



. Table 1 
1 

A Summary of the toefficient Procedures 
for the Validation of Learning Hierarchies 



Procedure , 



Proporjf^nt 



Proportion of Positive Transfer 

PPT Based Statistics • 
- Consistency Ratio 
Adequacy Ratio 
Completeness Ratio 
Inverse Consistency Ratio 
Inverse Adequacy Ratio 

tfon-PPT Based Statistics 
Phi Coefficient 
'Coefficient of Reproducibility 
m Proportion of Disconf irraatory 
Response Patterns 
Conditional Item Difficulty 



Gagne and Paradise (1961) 



Walbesser (1968) 



Walbesser and Eisenberg (1971) 



Capie ^nd Jones (1971) 

Hofmah (1977), Resnick and Wang (1964) 

Airasian and Bart (1975), 

Bart and Krus (1973) 
AiYasian (1971) 



To apply this model, it is assumed that the frequencies of the observed 

response patterns are? .distributed multinomially with probabilities given by 

the model. A test of the fit of the multincifial model (with the probabilities 

from the model as parameters) to the observed frequencies can be performed. 
• # 

White and Clark Model* A somewhat different mod^l was introduced by 
White and Clark (1973). This model tests the hypothesis that all- subjects 
who possess a certain skill form a subset of the group of subjects who 
possess a second skill. This model is called the C statistic model. 

In using tffis model, a matrix of response frequencies is developed 
using the format shown in Table 2. The population subgroups are defined 
as follows : . 



/ 



ii 



proportion of the population having,* 'neither skill, 

pro.p.ort/on of the population having both skills, 

proportion of the population having only Skill ,1,. and. 

proportion of the population having. only Skill II. 
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Table 2 

Response Frequency Matrix for White and Clark Model 



Number of Lower Skill 
Questions Correctly 
Answered 



Number of. Higher Skill 
Questions Correctly Answered 
0 1.2 TOTAL 



2 


P 20 


P 21 


P 22 


n 12 


. 1 




P ll 


P 12 


"11 


• 0 


p oo 


P 01 


P 02 


"10 


TOTAL 


V * n 20 




n 22 


N 



Note: N is the total number of subjects in the sample, while 
• n. . is the " 

for skill i 



n ij is the number of subjects correctly answering j items 



.The conditional probabilities of group members having answered the 
appropriate number of items correctly are: 

6 a = probability of examinee with Skill I answering correctly 
any item for Skill I, . y 

0 b = probability of examinee without Skill I answering correctly 
any Item for Ski.ll I, 6 y . 

© c - probability of examinee with Skill II answering correctlv » 
any item for Skill II, and 

e d = probability of examinee without Skill II answering correctlv 
any item for Skill II. .„ 6 y 

• * 

nrnhi h i-^ ta i P"; obabilit y of a «11 is thus Refined' as the product of the 
of memb r "of tha t ***** *' * ^ou^dthe conditional probability 

, b SOa 1 group answering the appropriate number of items correctly. • 

U Shi J 10 "' l * 3 ? d 0 5 are knOWn ' 3 Probability estimate for each cell 
usint tL r3? CalCUl3ted - Estimates of the P's .and 0's can be calculated 
using the cell frequencies or the marginal totals using a maximum likelihood 
procedure (see White and Clark, 1973). In a two-itef casfc , P 02 can be 
derived by substituting the estimates of P and 0 in the equation: 

P 02 " V 1 - V V + F,a - e a )V + p n « - o b ) 2 o c 2 + p b(1 - o a ) 2 o c 2 . (2) . 

of th^ C h P r ° babil i ties °l 311 ^ cells have been computed, the probability 
of the observed distribution under H Q can be calculated, using the observed • 

"sL f W-nn Cie A and h es ^ted probabilities as parameters of the multinomial 
distribution. A test of significance can be performed by summing the 



» 
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probabilities of all possible distributions which show a deviation from the 
hypothesis as great or greater than that of the observed distribution 
(White and Clark, 1973). In order to reduce the amount of computation 
required, White and Clark (1973) suggest forming the test using only the 
(0,2) cell. The observed frequency and estimated probability for this cell 
could be used as parameters for the binomial distribution, and H Q would be 

rejected whenever the observed frequency exceeded a critical value of C. 

The value of C, of course, 'depends on the desired error rate, the sample s 

size, and the magnitude of the probability estimated for the (0,2) cell. 

Dayton and Macready Model A third mathematical model for the validation 
of learning hierarchies was proposed by Dayton and Macready (1976). This ■ 
model is essentially a generalization of the Proctor model, and,.also subsumes 
the White and Clank model (Dayton and Macready, 1976). 

For any K dichotomously scored tasks, the scores on those tasks may be 
summarized by a column vector, U, composed of O's and. l f s. A score of 0 
may arise from an incorrgfct response or from an omission. The* product S = 
IT U.is the number of items or tasks successfully completed by a respondent. 
Assume there exists an a priori hierarchy , and there exists a set of q 
distinct pattern vectors, , comprised of O's and 2's, which for th£ 1 

hypothesized hierarchy defines acceptable response patterns. (The values 0 
and 2 are used* instead of 0 and 1 so that the vector difference V - U will 
provide information necessary to compute the values of the exponents in the 
model.) For example, when K = 4, a linear hierarchy would be represented 
by q = 5 pattern vectors: 



Vi.= 


(0000) , 


v 2 = 


(2ooo), 


v 3 = 


(2200), 


v 4 = 


(2220), 


v 5 = 


(2222). 



In the most general form the probabilistic model may be written a: 



P(u) 



7T a 

.1=1 




(3) 



where a..-, .b . . , c.., and d.. are defined as fbllows. Let g. . be. the ith 
ij ij' i] 1 ij • « , ij 

element in V. - U. Then, " 

J ... 
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a . . 



i if « - -i 



b. . 



c . . 



d. . 



1. if ty -sO 



1 if g.. = 2 
0 otherwise 



i if , .-i 



0 otherwise 



The parameter 0 j represents the probability that the jth pattern vector 
occurs. It is the hypothetical population proportion of respondents that 
achieves Level, j of the hierarchy. The parameter or. represents the probability 
that a respondent produces a correct response to a task, which relative* to 
a specific pattern vector should not have been correctly completed' The a 
parameter is referred to 'as the guessing parameter. The parameter p. is i 
the probability that a respondent produces a response that is incorrect 
which should have been completed correctly relative to a' specific pattern 
vector. The parameter p is the forgetting parameter. For this model, the 
occurrence of guessing and forgetting are assumed to be independent across 
items (Dayton and Macready, 1976). 



A restriction on 0. is 



q 

I 



0. = 1. 



Also, the parameters (0., Of.,, p.) are meaningful only over the interval 0 
to 1 (Dayton and Macready, 1976). 

For any given task, three of the four variables a?. . , b:.,. c \., and d.'. 
will be equal to. 0,' and one of the four will "take on the value of J l<. Thil^ 
model allows for 2 n - 1 independent parameters, where n is the number^ of' 
i-S'-u , P arameters have been estimated, .the Pearson chi-square or 

.likelihood ratio chi-square test may be used to provide a goodness-of-fit 
test (Dayton and Macready, 1976). 
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^ Loglinear Models Another class of models have been described by 
Goodman (1975) and Davisoi} (1981). These models are based or* an application 
of loglinear analysis to the problemtof^liscovering or confirming learning 
hierarchies. Loglinear analysis utilizes data in the form of a contingency 
table, but is not limited .to dichotomous variables, nor to pairwise comparisons 
('Davison, 19&1). For a discussion of loglinear models, see Bishop, Feinberg 
and- Holland (1975). % , \ 

For simplicity, the presentation of the model described by Goodman 
(1975) will be limited to the case of three variables, each having three 
response categories. The variables used could be items (for which the 
response categories would h£ item responses), or tests (for which the 
response categories would^oe test score categories) . The three variables 
and the£fc response categories form a contingency table, in J^is case, a 
three-way contingency table. Cell entries represent. frequencies of sub- 
jects exhibiting the response vector represented by each cell. That is, 
the entry in cell (a, b, c) would be th^/number of subjects responding a to 
the first variable, b to the second, and c to the third. 

Given an a priori or hypothesized, sequence among "the three variables, 
the contingency table can'be broken down into two classes of cell^: % those 
cells representing response vectors compatible with the hypothesized sequence, 
and those cells representing the vectors inconsistent with the hypothesized 
sequence. Subjects are divided into k»+ 1 classes, where k is the number 
of cells representing response vectors [compatible with the hypothesized 
sequence. The (k + l)th class is the iroup of subjects exhibiting incom- 
patible response vectors. The proportion of subjects in each class is 
represented by P. (i = 0, 1, . . . , WT, where Pq is the proportion in the 

class having inadmissible responses. 

Within the class of inadmissible cells, which is called the unsaleable 
class^ the variables are assumed to be independent, so that the probability 
of observing 9by given response* vector is simply the product of the probabil- 
ities of each of the responses to the. three variables. Tha(t is, P(a,b,c,) = 
P(a)P(b)P(c) for any unscaleable subject. Therefore, the joint probability 
of a randomly selected subject being in the unscaleable class and having 
response vector (a, b, c), is PQP(a)P(b)P(c) , regardless 6f whether cell* (a, 
b, c) is admissable or inadmissable . The probability of a subject from the 
scaleable class being in an inadmissible cell is zero. If (a, b, c) is aja 
admissible response vector, the probability of a scaleable subject having 
that response vector is P^, if (a, b, c) is the kth admissible cell. Of 

course, the probability of a subject from the kth class having a response 
vector represented by the jth cell, where j ^ k, is zero. This, leads to 
.the following equation, which is fundamental to the model: 



P(a, b, c) I 



0 ** s P^(a)P(b)P(c) if (a, b, c) is inadmissible 
P k + p 0 p ( a ) p ( b ) p ( c ) if ^ a > b > c ) is admissible^ 



This model is fitted to the data by one of several proposed algorithms 
(Goodman, 1975; Davison, 1980; Bishop, Fienberg and Hollard, 1975; Fienberg 
1977), and Pearson chi -square and likelihood ratio chi-square fit statistics 
can be obtained. 

< . ( . 

t 



ERIC 



-13- 



Summary of Model-Based Procedures A number of mathematical models for 
use in validating learning hierarchies have been presented. These procedures ■■ 

*re summarized in Table 3. Although there are four procedures listed in 
fable 3, those four procedures actually represent only two distinct types 
One type i>s represented by the Dayton and ^ac ready procedure, which subsumes 

^the Proctor, model and the White and Clark, njodel,. The other .type of preceding 
is represented by the Goodman and Davison model. • * 



Table 3 \ 

A Summary of the Procedures for the Validation 
of Learning Hierarchies Bas^i on Mathematical Models 



Procedure . Proponent 



Guttman Model Proctor (1970), 

C Statistic Model , . . White and Clark '197 D 

• - r • • ■ • 

- General Probabilistic Model Daytoil and Macready ' 9?6) - 

Loglinear Model '( Goodman (m5)> Davisor (198l) 



• Evaluation of Procedures for Validating Hierarchies 

The process of establishing a valid hierarchy involves ^u7ch more than the 
application of models and statistics " to the response patterns of individuals 
or the. mere application of Gagne's task analysis procedure. Before evaluating 
the procedures that have been proposed for the validation of learning 
hierarchies, A program in which the procedures should be applied will be 
discussed. Aiterward, the criteria to be used to evaluate the procedures 
within this cdntext will be presented, followed by an evaluation of the 
procedures. 

, Myogram for Constructi ng and Validating Hierarchies White (lW4a) 
has proposed a nine stage program for^he construction and validation of 
learning hierarchies that is comprehensive and directed at solving some of 
the problems encountered in the construction of learning hierarchies The ' 
nine stages in the program are: „ 

(a) Define in behavioral terms the element that is to be the highest 
stage of the hierarchy; 

(b) Derive the hierarchy by applying Gagne'-s task analysis procedure; 

lc) Check the reasonableness of the postulated hierarchy with experienced 
teachers and subject matter .experts^ 

(d) Invent possible divisions of the elements of the hierarchy, so 
that very precise definitions are obtained; 

(e) - Carry out an investigation of whether the invented divisions do 

in fact represent different skills; • - : . ~ ■ ' 
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(f) Write a learning .program for each element, and" embed in it mastery 
tests for each element; 

(g) Have at least 150 suitably chosen subjects work through the 
program, taking the tests as they come to them; 

-(h)' Analyze the results to see whether any of the postulated connections 
\ between elements should be rejected; and / 
(i) Remove all rejected connections from the hierarchy. ^ V 

Cotton, Gallagher, and Marshall (1977) have proposed a tenth stage 
which is to be used only if 15% or more of the connections were rejected. 
This stage would involve repeating stages (f) through (i), using the revised 
hierarchy with a different g^up of subjects^ 

The procedures that have been proposed for validating hierarchies are 
not employed until stage (h) of the White model. At this stage of the 
process a hierarchy has been postulated, and its reasonableness checked by 
experts and experienced teachers. What is needed at stage (h) is a procedure 
for validating the postulated hierarchical relationships between the elements 
of the hierarchy. The criteria for selecting such a procedure will now be 
discussed. 

Selection Criteria The first criterion for selecting a procedure for 
validating the hierarchical relationships between elements in a postulated 
hierarchy is that the procedure must provide information as to the direction 
of the relationship. Simply providing an indication of the strength of an 
association between elements is not sufficient. There could be a strong 
association between a subordinate and a superordinate element in a proposed 
hierarchy simply because the two elements measured the same skill. The 
strong association does not imply that the superordinate element could not f 
have been attained without prior success on the subordinate element. It 
must be demonstrated that success on the first element i-s not only sufficient, 
but a necessary condition for success on the second element. 

Another criterion is whether the procedure provides information useful 
for correcting the structure of the hierarchy if any connections are rejected 
as invalid. This includes information that would indicate Whether the 
ellements $re out of order, whether any unnecessary elements are included, 
and whether any necessary elements were omitted from the hierarchy. 

A third criterion for selecting a procedure for validating Ijigrarchies 
is whether the procedure indicates how successful a subject must be on an 
eljSfnent .before success is likely on the subsequent element. When a siiigle 
dithotomously scored item is used for each element, this is not an issue. 
But when each element involves a multi-item test, it is possible to reject 
a hierarchical relationship simply because success was poorly or incorrectly 
defined for that, element. " <* 

There are undoubtedly other criteria "that could be included in this . 
list ., 'However, these- should be adequate for evaluating the overall approaches 
toVvalidating learning hierarchies/ A detailed comparison of individual 
models^ with an eye toward selecting one or the other might require a more 
complete list of selection criteria. That is not the purpose here. The 
purpose .here is to evaluate the approaches to validating hierarchies that 
have been followed by researchers in order to make recommendations on the 
direction future research should follow. 
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Evaluation of the Coefficient Approach On the basis of the criteria 
for evaluation set out above, it must be concluded that the coefficient 
approariuto the validation of learning hierarchies is inadequate, file re 
are numerous problems with each of the coefficients, but a more serious"' 
roblem is the basic inadequacy of the approach itself. The first criterion 
or evaluation was that a validation procedure must indicate direction as 
[ell as, strength of a relationship. Coefficients such as those that have 
teen proposed for validating learning hierarchies do, not indicate direction. 
\m r instance > consider the case where N 00 is 20, N 0 i is 0, N 10 is 20, and 
is 60. For this case, the PPT statistic has a value of .80, CR = .75, 
1.0, COR = .75, and phi = :6l. If the two tasks are reversed, N 0 i is 
Sand N 10 is 0. These values yield a PPT s^ati^tic equal to 1.0', CR = 1.0, ■ 
^ .75, COR = .75, and phi = .61°. Reversing the two tables did not 
|ge the values of phi and, COR, and though it did' change the Values of 
* n PT and CR statistics, those statistics had high values regardless of 
"task was labeled Task 1. Clearly th^se statistics are not indicants 
direction of tKe relationship between the two tasks. 
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^efficients such as these also do not reliably indicate when unneccessary 
& are included in the hierafchy or necessary elements are omitted, 
eccessary elemeitf, is redundant, it will probably be correlated 
^her element or perhaps several elements in the hierarchy. Such a 

might result in high coefficient values and acceptance of the 
:y element. When a connection is rejected, c there is no indication 
her the 'connection was invalid or whether a' necessary element was 
ask 1 might be necessary but not sufficient fo.r attainment on 
|hout Task 2, even subjects successful on Task" 1 might be unable 
ccess on Task 3. In this case the hierarchy might be rejected 
being any indication that the hierarchy might have been 

Task 2 been included. The same situation might arise if 
eluded, but Tasks 2 and 3 were reversed in order.* 

|74b) has listed some other inadequacies of these coefficients, 
ies include the following: .because the sampling distributions 
" coefficients are unknown there is no way to determine the 
ion; an<}, these coefficients do not provide information 
ing success on multi-item tests. Based on White's criticisms 

it must be concluded 



Evaluatio! 
to the model-ba 



|tit approach and those listed above, 
coefficients for validating learning hierarchies does not 
|<kiuctive direction for future research. 

B. 

" ||the Model Approach Applying the criteria for evaluation 
procedures for validating hierarchies that have t&en 



proposed leads o^|&> conclude that , this is a very promising approach, but * 
that the flrocedu^^^ far developed .fall short of fully exploiting their 
potential. The^t^|£^e$ of models that have be$n proposed are a considerable 
improvement over t^||^ef f icient approach. When the fit of a particular 
model to the obser^l||4ata is satisfactory, then a reasonable amount of 
confidence can be in the validity of the hierarchy. Because the 

mathematical model ^lified in the procedure describes the way in Which 
performance on one ^Ifcat- in a hierarchy is related to performance on 
another element, theVife^ction of the relationship is indicated. Moreover, 
because the mathemati;®imodel is generally in the farm of a probability 
statement conditioned latent variable, the results are generalizable 

beyond the sample used; 1 ' 
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However, there are several problems with the models that have been 
proposed. One problem is that very little information is obtained for use 
in defining the criterion far success on multi-item tests. Another serious 
problem is encountered when the fit of the model to the observed data is 
inadequate. When the model is rejected,. it is not easy to determine why it 
was rejected. A -model could be rejected because the hierarchy is incorrectly 
specified, because the mathematical modelr is of an inappropriate form, or 
both, tig/inf ormation is provided as to whether unnecessary\elements were 
included or necessary elements were omit,ted\ Because of thefee inadequacies, 
the model-bas.6d procedures that have 1 been proposed need improvement. More 
research on this approach is needed. 

Summary and Recommendations 

Summary The area of individualized instruction was identified as one 
of the fastest growing areas in\the field of education. One of the most 
crucial components of individualized instruction was found to be the sequencing 
of instructional units. A review of the literature was undertaken to 
identify the major alternatives available for sequencing unfts of instruction 
in such a way as to facilitate education. 

- Procedures for validating sequences <5f instructional units, or learning 
hierarchies, were found to fall into two general categories. One category 
included procedures based on coefficients of dependence, while the otfter 
category contained procedures based on more complete descriptions of the 
relationships between units of instruction , usually a mathematical model. 
An evaluation of these two approaches to validating learning hierarchies 
was undertaken in order to facilitate the formulation of recommendations 
for the direction of future research in this area. 

Procedures based on coefficients were found to provide insufficient 
information for the validation of hierarchies or for, correcting deficiencies 
in the structure of a proposed hierarchy. Procedures based on mathematical 
mqdels were also 1 found to be* inadequate, but the model-based approach was 
found to have potential for dealing with the problem of learning hierarchy 
validation. Recommendations • for future research follow. 

Recommendations In future research on the' validation of learning 
hierarchies, emphasis should be placed on improving the* currently available 
model-based procedures in several ways. A validation procedure should 
provide information useful for defining success on the elements of a learning 
hierarchy, as well as information as to whether important elements have 
been omitted, .inappropriate elements have been included/or some elements 
have ju9t been placed in the wrong order. "Developing a procedure that will 
provide all this information will make it possible not just to accept or 
reject a hierarchy, but to correct deficiencies in the structure of rejected 
hierarchies. It appears that the development of such a procedure would 
greatly facilitate future development of the field of individualized 
instruction. 

\ 

/ 
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